Prompt Generator for Stable Diffusion#

Note

Install ekorpkit package first.

Set logging level to Warning, if you don’t want to see verbose logging.

If you run this notebook in Colab, set Hardware accelerator to GPU.

!pip install ekorpkit[art] exit()

Preparing the environment#

%load_ext autotime
%load_ext autoreload
%autoreload 2 # reload modules before executing user code, set to 0 to disable

from ekorpkit import eKonf

eKonf.setLogger("INFO")
eKonf.set_cuda(device=4)
print("version:", eKonf.__version__)

is_notebook = eKonf.is_notebook()
is_colab = eKonf.is_colab()
print("is notebook?", is_notebook)
print("is colab?", is_colab)
if is_colab:
    eKonf.mount_google_drive()
eKonf.set_workspace(workspace="/content/drive/MyDrive/workspace/", project="ekorpkit-book")

INFO:ekorpkit.base:Setting cuda device to ['A100-SXM4-40GB (id:4)']
INFO:ekorpkit.utils.notebook:shell type: ZMQInteractiveShell
INFO:ekorpkit.utils.notebook:Google Colab not detected.
INFO:ekorpkit.base:Setting EKORPKIT_WORKSPACE_ROOT to /content/drive/MyDrive/workspace/
INFO:ekorpkit.base:Setting EKORPKIT_PROJECT to ekorpkit-book
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env

version: 0.1.40.post0.dev3
is notebook? True
is colab? False

('/content/drive/MyDrive/workspace', 'ekorpkit-book')

time: 2.29 s (started: 2022-11-02 08:08:26 +00:00)

Preparing the dataset#

from ekorpkit.models.art.prompt import PromptGenerator

pgen = PromptGenerator(verbose=False)

2022-11-02 08:08:36.692394: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO:ekorpkit.batch:Batch name: prompt-batch, Batch num: 14

time: 3.93 s (started: 2022-11-02 08:08:35 +00:00)

prompts = pgen.generate_prompts(prompt="island")
prompts

INFO:ekorpkit.batch:Using existing path: /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen
INFO:ekorpkit.batch:Batch name: prompt-batch, Batch num: 14
INFO:ekorpkit.batch:Merging config with args: {}
INFO:ekorpkit.batch:Setting seed to 1075645782
INFO:ekorpkit.models.art.prompt:Loading model from /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts
INFO:ekorpkit.batch:Saving config to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-prompt/prompt-batch/prompt-batch(14)_config.yaml

['islanders, hyperdetailed, digital painting, concept art, sharp focus, illustration, fantasy world, photorealistic, masterpiece a portrait of a woman in a sci - fi outfit with cybernetic implants, cyberpunk tech wear',
 'island, jossi riabovitchev, marc simonetti, yoshitaka amano 3 d goddess close - up profile, symmetrical perfect face, flowing long hair, dazzling light orbs, highly',
 'island 2 4 5 3 concept art oil painting of a lushly lit cityscape with lush vegetation and stunning voluptuous ethereal pink pink neon flowers, stunning voluptuous nordic imagery, voluptuous oriental art nouve',
 'island, volumetric lighting, red and cyan theme, artstation, concept art by craig mullins dnd character portrait concept art, dnd, rpg character, trending on artstation, digital art, by studio',
 'island, concept art, matte painting, fantasy, sharp focus, vibrant colors A full portrait of a beautiful ethereal kat dennings by Studio Ghibli, Makoto Shinkai, by Artgerm, by W',
 'island - kalachakra, full body shot, white hair, red hair, brown eyes, white narrow mid cut shirt, jeans, large bellybutton, high waist, dramatic light, cinematic wallpaper, anime, ilya kuvshin',
 'islandland, new research notes, post grunge, concept art by peter elson johannes voss molly millions, dark fantasy magic, concept art, trending on art station, award-winning photograph, masterpiece',
 'island on fire in las vegas, epic landscape, detailed landscape, arctic, fantasy magic, fairytale, storyteller, atmospheric, intricate, headshot, highly detailed, digital painting, artstation, concept art, sharp focus',
 'island, concept art, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha, footage from space camera',
 'island, artstation, cgsociety, concept art A group portrait of cute Incredibly beautiful cute Blackpink Lalisa Manoban wearing fantastic dress in the style of Assamese bihu mekhela sad']

time: 18.6 s (started: 2022-11-02 08:08:41 +00:00)

pgen.train(model_name="ekorpkit/stable-prompts-2")

WARNING:datasets.builder:Using custom data configuration Gustavosta--Stable-Diffusion-Prompts-d22aeec0ba2a9fdb
WARNING:datasets.builder:Reusing dataset parquet (/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache/Gustavosta___parquet/Gustavosta--Stable-Diffusion-Prompts-d22aeec0ba2a9fdb/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)

Token indices sequence length is longer than the specified maximum sequence length for this model (2613888 > 1024). Running this sequence through the model will result in indexing errors
***** Running training *****
  Num examples = 18847
  Num Epochs = 1
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 2356
Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
ERROR:wandb.jupyter:Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: Currently logged in as: entelecheia. Use `wandb login --relogin` to force relogin
wandb version 0.13.4 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.13.3
Run data is saved locally in /workspace/projects/ekorpkit-book/ekorpkit-book/docs/lectures/aiart/wandb/run-20221102_081028-3heixjgo
Syncing run ekorpkit/stable-prompts-2 to Weights & Biases (docs)
[2356/2356 04:59, Epoch 1/1]
Step Training Loss
200 3.776300
400 2.705100
600 2.517600
800 2.376600
1000 2.295400
1200 2.231800
1400 2.175900
1600 2.143700
1800 2.104900
2000 2.082700
2200 2.081700 

Training completed. Do not forget to share your model on huggingface.co/models =)
Waiting for W&B process to finish... (success).

Run history:


train/epoch▁▂▂▃▄▄▅▆▆▇▇█
train/global_step▁▂▂▃▄▄▅▆▆▇▇█
train/learning_rate█▇▇▆▅▄▄▃▂▂▁
train/loss█▄▃▂▂▂▁▁▁▁▁
train/total_flos▁
train/train_loss▁
train/train_runtime▁
train/train_samples_per_second▁
train/train_steps_per_second▁

Run summary:


train/epoch1.0
train/global_step2356
train/learning_rate0.0
train/loss2.0817
train/total_flos4924659858407424.0
train/train_loss2.38621
train/train_runtime353.5289
train/train_samples_per_second53.311
train/train_steps_per_second6.664

Synced ekorpkit/stable-prompts-2: https://wandb.ai/entelecheia/ekorpkit-book-prompt-generator/runs/3heixjgo
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20221102_081028-3heixjgo/logs
INFO:ekorpkit.models.art.prompt:Saving model to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2
Configuration saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/config.json
Model weights saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/pytorch_model.bin
INFO:ekorpkit.batch:Saving config to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-prompt/prompt-batch/prompt-batch(14)_config.yaml

time: 7min 9s (started: 2022-11-02 08:09:17 +00:00)

pgen.load_model(model_name="ekorpkit/stable-prompts-2")

INFO:ekorpkit.models.art.prompt:Loading model from /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2
loading configuration file /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/config.json
Model config GPT2Config {
  "_name_or_path": "/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2",
  "_num_labels": 1,
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": null,
  "n_layer": 6,
  "n_positions": 1024,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.1,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50
    }
  },
  "torch_dtype": "float32",
  "transformers_version": "4.21.2",
  "use_cache": true,
  "vocab_size": 50260
}

loading weights file /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/pytorch_model.bin
All model checkpoint weights were used when initializing GPT2LMHeadModel.

All the weights of GPT2LMHeadModel were initialized from the model checkpoint at /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.

time: 1.42 s (started: 2022-11-02 08:04:02 +00:00)

Generate images#

text_prompts = "a living tiger inside a gorgeous cave in a futuristic Jeju island, relaxing night, watercolour on the ceiling, glowing on glass, light art, dark, dark, gorgeous, dark, bright in an abandoned village in a futuristic alien nightscape"
batch_name = "cave-jeju"